Systems and methods for pseudo image data augmentation for training machine learning models

ABSTRACT

Systems and methods for augmenting a training data set with annotated pseudo images for training machine learning models. The pseudo images are generated from corresponding images of the training data set and provide a realistic model of the interaction of image generating signals with the patient, while also providing a realistic patient model. The pseudo images are of a target imaging modality, which is different than the imaging modality of the training data set, and are generated using algorithms that account for artifacts of the target imaging modality. The pseudo images may include therein the contours and/or features of the anatomical structures contained in corresponding medical images of the training data set. The trained models can be used to generate contours in medical images of a patient of the target imaging modality or to predict an anatomical condition that may be indicative of a disease.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/835,033, filed Mar. 30, 2020, which is incorporated herein byreference in its entirety.

FIELD

The present disclosure relates generally to medical imaging, and moreparticularly, to systems and methods for augmenting a training data setwith annotated pseudo images for training machine learning models.

BACKGROUND

Radiotherapy is an important tool for the treatment of cancerous tumorsin patients. Unfortunately, ionizing radiation applied to treat thepatient does not inherently discriminate between tumors and proximalhealthy structures (e.g., organs-at-risk). Administration of theionizing radiation thus must be carefully tailored to restrict theapplied radiation to the target (i.e., tumor) while avoiding unnecessaryirradiation of surrounding anatomy, the goal being to deliver a lethalradiation dose to the tumor while maintaining an acceptable dosage tothe proximal structures.

As part of the radiotherapy planning process, medical images of thetumor and surrounding anatomy are obtained. The medical images can serveas a basis for simulations of the radiation treatment and can be used toplan various aspects of the therapy, including but not limited to, beamgeometry and location, radiation energy, and dosage. The medical imagesare typically processed to delineate target regions (e.g., pixels orvoxels where a tumor or other regions desired to be irradiated areimaged) and separate surrounding structures (e.g., pixels or voxelswhere an organ-at-risk (OAR) or other anatomical structure to avoidbeing irradiated is imaged). This delineation, termed contouring orsegmenting, involves defining a respective border defining outlines ofthe different anatomical structures in the image. However, if anatomicalstructures are improperly contoured in the images, this could result ininsufficient irradiation of the target and/or undesirable irradiation ofsurrounding structures.

Manual contouring of structures in medical images can be atime-consuming phase in the radiotherapy planning process. To addressthis issue, automatic segmentation models, such as machine learningmodels, have been proposed. Machine learning generally involvesextracting feature vectors from images, such as for each voxel, etc.,that may be used as input to a machine learning model, such as neuralnetworks, random forests, probabilistic models, support vector machines,and dictionary learning, for example, that classify which class eachvoxel belongs to.

Generally, machine learning methods involve a training phase and aninference phase. In the training phase, a machine learning model usestraining data sets of medical images to generate a particular output.For example, the training data set can include 2-D or 3-D images withground truth contours for the anatomical structures imaged by thedifferent pixels or voxels. Training of the machine learning modelinvolves the iterative process of adjusting weighting values, generallydetermined during the training process, until an input results in adesired output, at which point the machine learning model may beconsidered “trained.”

During the inference phase, the trained machine learning model operateson medical image(s) of a patient to automatically process features ofthe medical image(s) to generate a particular output, such as contourdata of specific anatomical structures in the medical image(s).

Since machine learning based methods are strongly dependent on the dataset they train on, in order to accurately train a machine learningmodel, a large amount of consistently labelled (annotated) training datais required. This, however, may be problematic, especially for imagingmodalities where obtaining accurately labelled training data isdifficult.

For example, for imaging modalities that are not conventionally used fortreatment planning, but which are used for other aspects and/or otherphases of the radiotherapy treatment process (i.e., used to detect dailyvariations in patient anatomy and/or for patient positioning, forexample), consistently labelled image training data may not be availablefrom the clinical routine. One of the reasons for lack of such trainingdata is because labelling of such images is very difficult due to theimage quality. In such cases, segmentations are generally done on imagesof a different imaging modality that produce better image quality, andthe contours are transferred to the planning images. Transfer betweenimaging modalities, however, may require image registration that may adduncertainty to the segmentation.

Another reason for lack of such training data is because images may notbe readily available due to lack of clinical implementation of theimaging technique for a specific treatment region, lack of availabilityof a newly developed imaging equipment, or limited access to newlydeveloped reconstruction algorithms of imaging sequences.

Cone-beam computed tomography (CBCT) is an imaging modality that servesas an example of both problems. CBCT images are not segmented routinely,and therefore are not readily available as training data. Even if theywere, due to different types of artifacts, including noise, beamhardening, and scattering, for example, that are present in this imagingmodality, the segmentations obtained in such images contain largeuncertainties due to decreased image contrast.

As illustrated in FIGS. 1A and 1B, for example, CBCT scans are generallyknown to have poor soft tissue contrast. As shown in FIG. 1A, forexample, in the CBCT scan 1A, there is no clear separation between thebowel tissue and the surrounding fat, whereas in the CT scan 1B of thesame abdominal region, as shown in FIG. 1B, the separation is muchclearer. This makes it very difficult for an observer to accuratelydelineate anatomical structures on CBCT scans. As a result, it isdifficult to generate a large set of consistently labelled CBCT trainingdata. Training neural networks with such scarce and/or uncertain data,however, can potentially negatively affect the training process and theobtained results.

Although data augmentation is a process that is commonly used toartificially increase the size of a training data set by generatingadditional training data from the existing one, applying conventionaldata augmentations, such as data warping or oversampling, for example,on inaccurately labelled training images will only provide additionalinaccurately labelled images. On the other hand, applying conventionaldata augmentation on a training data set of a different imagingmodality, such as computed tomography (CT) scans, for example, will notprovide accurate segmentation data for CBCT scans, since the augmentedCT scans do not reproduce appearance that is typical for a CBCT targetimaging modality, such as artifacts that are typical for the CBCTimaging modality.

Embodiments of the disclosed subject matter may address one or more ofthe above-noted problems and disadvantages, among other things.

SUMMARY

Embodiments of the disclosed subject matter provide systems and methodsfor augmenting a training data set with pseudo (synthetic) images thatprovide a realistic model of the interaction of image generating signalswith the patient, while also providing a realistic patient model.

Embodiments of the disclosed subject matter provide systems and methodsfor training a machine learning model using the augmented training dataset.

Embodiments of the disclosed subject matter provide systems and methodsfor training a machine learning model using the augmented training dataset for feature learning and/or pattern analysis and/or classificationand/or segmentation.

In exemplary embodiments, the machine learning model is one of asegmentation model and a diagnostic model, the segmentation model beingtrained to generate contours in medical images, and the diagnostic modelbeing trained to detect a feature representative of a medical conditionin the medical images and to predict a condition, such as the presenceof a disease, from the medical images.

In exemplary embodiments, the segmentation model is a neural networkmodel, and the diagnostic model is one of a random forest classifier, aprobabilistic model, a dictionary learning model, and a support vectormodel.

Embodiments of the disclosed subject matter provide systems and methodsfor training a deep learning based neural network model (DNN) using theaugmented training data set.

Embodiments of the disclosed subject matter further provide systems andmethods for training the deep learning based neural network model (DNN)using the augmented training data set to generate contours on images ofa target imaging modality.

Embodiments of the disclosed subject matter further provide systems andmethods for training the diagnostic model (i.e., one of a random forestclassifier and/or the probabilistic model and/or support vector model,for example) using the augmented training data set to detect thepresence of anatomical abnormalities and/or diseases in medical images.

In exemplary embodiments the augmented training data set includes anoriginal training data set and a supplemental training data set, thesupplemental training data set including pseudo (synthetic) imagesgenerated from the original training data set.

Embodiments of the disclosed subject matter further provide systems andmethods for augmenting a training data set of a first imaging modalitywith pseudo (synthetic) images that provide realistic image andsegmentation data for a target imaging modality. The target imagingmodality may be different from the first imaging modality.

Embodiments of the disclosed subject matter further provide systems andmethods for training a segmentation DNN model to generate contours onimages of a target imaging modality using a training data set thatincludes an original training data set and a supplemental training dataset generated from the original training data set, the supplementaltraining data set providing realistic image and segmentation data forthe target imaging modality.

Embodiments of the disclosed subject matter further provide systems andmethods for training a machine learning algorithm to detect the presenceof abnormalities in images of a target imaging modality using a trainingdata set that includes an original training data set and a supplementaltraining data set generated from the original training data set, thesupplemental training data set providing realistic image and diagnosticdata for the target imaging modality.

Embodiments of the disclosed subject matter further provide systems andmethods for training a machine learning model to detect the presence ofan abnormality and/or disease in images of a target imaging modalityusing a training data set that includes an original training data setand a supplemental training data set generated from the originaltraining data set, the supplemental training data set providingrealistic image and diagnostic data for the target imaging modality.

In exemplary embodiments, the supplemental training data set includespseudo (synthetic) images generated from the original training data set.

In exemplary embodiments, the original training data set includesoriginal training images of a first imaging modality, and the generatedpseudo images include images of a second (target) imaging modality.

In exemplary embodiments, the pseudo images contain the same groundtruths as the original training images.

In exemplary embodiments, the pseudo images contain contours as theground truth.

In exemplary embodiments, the pseudo images contain features associatedwith a disease and/or anatomical abnormality as the ground truth.

Embodiments of the disclosed subject matter also provide systems andmethods for generating the pseudo images.

In exemplary embodiments, the generating of the pseudo images includesat least a forward projection step or a forward and backward projectionstep to generate projection images from images of the original trainingdata set, and to reconstruct the projection images into a pseudo 3Dvolume or pseudo 2D scans, taking into consideration radiation doseattenuation and reconstruction artifacts specific to the target imagingmodality.

Alternatively, the generating of the pseudo images includes using aneural network trained to predict a pseudo image of a target imagingmodality based on images of a first imaging modality.

Embodiments of the disclosed subject matter also provide for a systemcomprising: one or more data storage devices storing at least one neuralnetwork model, the neural network model having been trained toapproximate a contour of an anatomical structure; and one or moreprocessors operatively coupled to the one or more data storage devicesand configured to employ the at least one neural network model toprocess one or more medical images of a patient of a target imagingmodality to generate one or more contours of anatomical structures inthe medical images of the patient, wherein the one or more processorsare further configured to train the neural network model to approximatecontours of anatomical structures using a first data set of medicalimages and a second data set of medical images, the medical images ofthe first data set including medical images of a first imaging modality,and the medical images of the second data set including pseudo medicalimages of the target imaging modality, the one or more processors beingfurther configured to generate the pseudo medical images of the seconddata set from corresponding medical images of the first data set.

Embodiments of the disclosed subject matter also provide for a systemcomprising: one or more data storage devices storing at least onemachine learning model, the machine learning model having been trainedto detect the presence of abnormalities and/or disease; and one or moreprocessors operatively coupled to the one or more data storage devicesand configured to employ the at least one machine learning model toprocess one or more medical images of a patient of a target imagingmodality to generate a diagnostic outcome, wherein the one or moreprocessors are further configured to train the machine learningalgorithm to detect the presence of abnormalities and/or disease using afirst data set of medical images and a second data set of medicalimages, the medical images of the first data set including medicalimages of a first imaging modality, and the medical images of the seconddata set including pseudo medical images of the target imaging modality,the one or more processors being further configured to generate thepseudo medical images of the second data set from corresponding medicalimages of the first data set.

Embodiments of the disclosed subject matter also provide for anon-transitory computer-readable storage medium upon which is embodied asequence of programmed instructions, and a computer processing systemthat executes the sequence of programmed instructions embodied on thecomputer-readable storage medium to cause the computer processing systemto train a neural network model to approximate contours of anatomicalstructures using a first data set of medical images and a second dataset of medical images, the medical images of the first data setincluding medical images of a first imaging modality, and the medicalimages of the second data set including pseudo medical images of asecond imaging modality, generate the pseudo medical images of thesecond data set from corresponding medical images of the first data set;and process one or more medical images of a patient using the trainedneural network model to generate one or more contours of anatomicalstructures in the medical images of the patient.

Embodiments of the disclosed subject matter also provide for anon-transitory computer-readable storage medium upon which is embodied asequence of programmed instructions, and a computer processing systemthat executes the sequence of programmed instructions embodied on thecomputer-readable storage medium to cause the computer processing systemto train a machine learning model to detect abnormalities and/or diseaseusing a first data set of medical images and a second data set ofmedical images, the medical images of the first data set includingmedical images of a first imaging modality, and the medical images ofthe second data set including pseudo medical images of a second imagingmodality; generate the pseudo medical images of the second data set fromcorresponding medical images of the first data set; and process one ormore medical images of a patient using the trained neural network modelto generate a diagnostic output for the patient.

In exemplary embodiments, the execution of the sequence of programmedinstructions can further cause the computer processing system togenerate the pseudo images of the second imaging modality using one of aforward projection technique, a forward projection coupled with abackward projection technique, and a trained neural network technique,wherein the forward projection technique includes generating projectionimages from the medical images of the first data set by generating avolumetric image from the first set of medical images and simulatingradiation passing through the volumetric image, wherein the backwardprojection technique includes reconstructing the projection images intoa pseudo volumetric image of the medical images of the second data set,wherein the reconstructing of the projection images includesaccumulating the projection images onto a pseudo 3D volume, andgenerating the pseudo medical images of the second data set from thepseudo volumetric image, and wherein the neural network techniqueincludes using a second neural network trained to predict a pseudo imageof a second imaging modality based on an image of a first imagingmodality.

Objects and advantages of embodiments of the disclosed subject matterwill become apparent from the following description when considered inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments will hereinafter be described with reference to theaccompanying drawings, which have not necessarily been drawn to scale.These drawings are for illustration purposes only and are not intendedto limit the scope of the present disclosure in any way. Whereapplicable, some features may not be illustrated to assist in theillustration and description of underlying features. Throughout thefigures, like reference numerals denote like elements. As used herein,various embodiments can mean one, some, or all embodiments.

FIGS. 1A-1B are illustrations of exemplary CBCT and CT scans of anabdominal region of a patient.

FIG. 2A is a simplified schematic diagram of operation of a neuralnetwork model during a training phase, according to various embodimentsof the disclosed subject matter.

FIGS. 2B-2C is a simplified schematic diagram of operation of a neuralnetwork model during an inference phase, according to variousembodiments of the disclosed subject matter.

FIG. 2D is a process flow diagram for generalized training and inferencephases of a neural network model, according to various embodiments ofthe disclosed subject matter.

FIG. 3 is a simplified node map of a deep neural network, according tovarious embodiments of the disclosed subject matter.

FIG. 4 is a process flow diagram for details of the training phase of aneural network model, according to various embodiments of the disclosedsubject matter.

FIG. 5 a simplified schematic diagram illustrating aspects of a medicalimage processing systems, according to various embodiments of thedisclosed subject matter.

FIG. 6 illustrates aspects of a radiation therapy system, according tovarious embodiments of the disclosed subject matter.

FIG. 7 is a simplified schematic diagram of structure and dose dataobtained during radiotherapy treatment processing, according to variousembodiments of the disclosed subject matter.

FIG. 8 is a process flow diagram for generalized training and inferencephases of a neural network model, according to various embodiments ofthe disclosed subject matter.

FIG. 9 is a process flow diagram for details of the training phase ofthe neural network model, according to various embodiments of thedisclosed subject matter.

FIGS. 10A-10D are simplified schematic diagrams of operation of a neuralnetwork model during a training phase, according to various embodimentsof the disclosed subject matter.

FIGS. 11A and 11B are simplified schematic diagrams of operation of aneural network model during a training phase, according to variousembodiments of the disclosed subject matter.

FIGS. 12A-12C are simplified schematic diagrams of operation of a neuralnetwork model during an inference phase, according to variousembodiments of the disclosed subject matter.

FIG. 13 is a process flow diagram for generalized pseudo imagegeneration processes, according to various embodiments of the disclosedsubject matter.

FIGS. 14-16 are simplified schematic diagrams of projection imagegeneration, according to various embodiments of the disclosed subjectmatter.

FIG. 17 is a simplified schematic diagram of pseudo image reconstructionfrom the projection images, according to various embodiments of thedisclosed subject matter.

FIG. 18 is a simplified schematic diagram of a neural network modeltraining based on projection image data, according to variousembodiments of the disclosed subject matter.

FIG. 19 is a simplified schematic diagram of the trained neural networkmodel during an inference phase, according to various embodiments of thedisclosed subject matter.

FIG. 20 is a simplified schematic diagram of operation of a trainedneural network model to generate pseudo images, according to variousembodiments of the disclosed subject matter.

FIGS. 21A-21B are simplified schematic diagrams of operation of adiagnostic model during a training phase, according to variousembodiments of the disclosed subject matter.

FIG. 22 is a simplified schematic diagram of operation of a diagnosticmodel during a training phase, according to various embodiments of thedisclosed subject matter.

DETAILED DESCRIPTION

In operation, a machine learning based model involves a training phase(TP) and an inference phase (IP). In the training phase, the machinelearning model uses training data sets of medical images to generate aparticular output (e.g., generate contours, detect anatomic anomalies,etc.), and in the inference phase, the trained model operates on amedical image set containing medical images of a new patient toautomatically process learned features of the medical image(s), such asdetermining contours of unknown anatomical structures in the image(s),and/or detect anatomic anomalies, and/or features relating to a specificdisease and/or to classify the detected anomalies/features, etc.

Operation of a machine learning model for segmentation, such as a deepneural based (DNN) segmentation model is shown in FIGS. 2A-2D, forexample. As used herein, the terms “deep learning model” or “deep neuralnetwork model” refer to a class of computer-based machine-learningalgorithms that utilize many layers or stages (in particular, at leasttwo “hidden” layers between input and output layers) of data processingfor feature learning, pattern analysis, and/or classification. Ingeneral, these DNN models are formed by a layered network of processingelements (referred to as neurons or nodes) that are interconnected byconnections (referred to as synapses or weights). The layers of nodesare trained from end-to-end (i.e., from input layer to output layer) toextract feature(s) from the input and classify the feature(s) to producean output (e.g., classification label or class).

FIG. 3 illustrates a simplified node map 250 for an exemplary DNN model.The DNN model includes a stack of distinct layers (vertically orientedin FIG. 3 ) that transform an input (provided to the nodes 258 of inputlayer 252) into an output (at nodes 262 of output layer 256). Theintervening layers (Layer 1 through Layer n) between the input layer 252and output layer 256 are referred to as “hidden” layers 254. At leasttwo hidden layers are provided in order for the neural network to beconsidered “deep.” Each hidden layer has respective nodes 260, whichperform a particular computation and are interconnected to nodes inadjacent layers. For example, each node 260 can include a weightingfunction, which provides weights to respective inputs, and an activationfunction, which processes the weighted inputs to generate respectiveoutputs. The different hidden layers 254 can include, but are notlimited to, final loss layers, non-linear operator layers, poolinglayers, subsampling layers, upsampling layers, fully connected layers,and convolutional layers. Although FIG. 3 illustrates the hidden layers254 as having more nodes 260 per layer than a number of the nodes258/262 in the input 252 and output 256 layers, other numbers andconfigurations are also possible. The simplified map illustrated in FIG.3 is intended to be exemplary only, and other maps based on a selectedDNN (e.g., a convolutional neural network) are also possible accordingto one or more contemplated embodiments.

In the training phase (TP), the segmentation DNN model 20 uses trainingdata sets 10 of medical images 11 to generate a particular output 21.For example, the training data set 10 can include two-dimensional (2-D)or three-dimensional (3-D) images 11 with ground truth contours 12 forthe anatomical structures imaged by the different pixels or voxels. Fortraining of the DNN model 20, the training data set 10 can includeadditional ground truth information, such as cut-off plane locationand/or user-defined ROIs (e.g., bounding boxes), for example. As usedherein, “training” refers to determining one or more parameters of nodesin hidden layers of the DNN model 20, for example, by an iterativeprocess S100 illustrated in FIG. 4 , that varies parameters such thatthe DNN model 20 output 21 more closely matches corresponding groundtruth. For example, as shown in FIG. 3 , nodes 260 in the hidden layer254 can include a filter or kernel, parameters of which (e.g., kernelweight, size, shape, or structure) can be adjusted during the trainingprocess.

FIG. 4 illustrates the iterative model training process S100. In stepS102, the training data 10 supplied in S101 is propagated through thenodes of hidden layers of an input DNN model. The resulting data fromthe hidden layer are provided to nodes of the output layer of the DNNmode in S103. In step S104, the data from the output layer is comparedwith the ground truth via a loss function 22. For example, loss function22 can be mean-squared error, dice loss, cross entropy-based losses orany other loss function known in the art.

During the training, the DNN model is given feedback by loss function 22on how well its output matches the correct output. Once an iterationcriteria is satisfied at S105 (e.g., loss function meets a predeterminedthreshold, a threshold number of iterations has been reached, or nofurther improvement is seen between iterations), the DNN model is fixedat S107. Otherwise, the training S100 proceeds to S106, where the DNNmodel is modified, e.g., by adjusting parameters of the hidden layernodes, in order to improve the match between output and the desiredoutput. The training process S100 can iterate repeatedly until thedesired iteration criteria is met at S105. The DNN model is thenconsidered trained and the trained DNN model of S107 can be stored in animage segmentation model database.

During the inference phase (IP), the trained DNN model 20 can operate ona medical image set 30 containing medical images 31 of a new patient toautomatically process features of the medical image(s) 31, such asdetermining contours 40 of unknown anatomical structures in the image(s)31. The contoured image data set 50 may then be used to generate atreatment plan for the patient.

Each respective DNN model may run on a corresponding DNN engine, whichrefers to any suitable hardware and/or software component(s) of acomputer system that is capable of executing algorithms according to anysuitable deep learning model. In embodiments, the deep learning model(s)can be based on any existing or later-developed neural network, orcombinations thereof. Exemplary neural networks include, but are notlimited to, a convolutional neural network (ConvNet or CNN) (e.g.,U-Net, deep CNN, LeNet, V-Net, AlexNet, VGGNet, Xception, DenseNet,GoogLeNet/Inception, etc.), residual neural network (ResNet), recurrentneural network (RNN) (e.g., Hopfield, Echo state, independent RNN,etc.), long short-term memory (LSTM) neural network, recursive neuralnetwork, generative adversarial neural networks (GANs), normalizingflows and graph networks, and deep belief network (DBN).

To generate the medical images (whether 2-D or 3-D) of the training sets10 and/or of the patient set 30, any suitable medical imaging modalityor modalities can be used, such as, but not limited to, X-ray, computertomography (CT), cone beam computed tomography (CBCT), spiral CT,positron emission tomography (PET), magnetic resonance imaging (MRI),functional MRI, single photon emission computed tomography (SPECT),optical tomography, ultrasound imaging, fluorescence imaging,radiotherapy portal imaging, or any combinations thereof. For example,image data may include a series of 2-D images or slices, eachrepresenting a cross-sectional view of the patient's anatomy.Alternatively, or additionally, image data may include volumetric or 3-Dimages of the patient, or a time series of 2-D or 3-D images of thepatient.

As discussed above, for certain imaging modalities, training asegmentation model to generate contours in patient images is difficult,because there is either no accurate training data available, or becauseaugmenting the available training data does not provide both realisticimage and segmentation data. In particular, augmented data obtainedthrough the conventional augmentation processes does not reproducefeatures which are typical for the image acquisition mode or acquisitiondevice used for imaging the patient, and therefore does not provide arealistic (i.e., physical) model of the interaction of image generatingsignals with the patient while also providing a realistic patient modelcreated from the original training data set.

FIG. 5 illustrates aspects of a computer infrastructure 310 that canprovide a solution to one or more of these issues. For example, thecomputer infrastructure 310 is configured to augment an originaltraining data set with pseudo (synthetic) images that provide arealistic model of the interaction of image generating signals with thepatient, while also providing a realistic patient model.

The computer infrastructure 310 is further configured to train machinelearning based models 315 (i.e., training phase), including, but notlimited to, segmentation models (segmentation DNNs) and diagnosticmodels/classifiers (random forest, support vector, etc.), to generatecontours on patient images and/or detect and/or to predict anatomicanomalies and/or diseases (i.e., inference phase) using a training dataset that is augmented with pseudo (synthetic) images that provide arealistic model of the interaction of image generating signals with thepatient, while also providing a realistic patient model.

The computer infrastructure 310 can further provide for training of asegmentation DNN model 315 to generate contours on images of a targetimaging modality using a training data set that includes an originaltraining data set of a first imaging modality and a supplementaltraining data set of a second imaging modality generated from theoriginal training data set, the supplemental training data set providingrealistic image and segmentation data for the target imaging modality.The supplemental training data set includes pseudo (synthetic) images(scans) generated from the original training data set, which contain thesame ground truth contours as the original training images.

The computer infrastructure 310 can further provide for training of adiagnostic classifier 315 to detect and/or predict anatomic anomaliesand/or diseases from images of a target imaging modality using atraining data set that includes an original training data set of a firstimaging modality and a supplemental training data set of a secondimaging modality generated from the original training data set, thesupplemental training data set providing realistic image and patientdata for the target imaging modality. The supplemental training data setincludes pseudo (synthetic) images (scans) generated from the originaltraining data set, which contain the same ground truths as the originaltraining images.

The computer infrastructure 310 can also provide procedures to develop,train and utilize artificial intelligence (AI) or other processingtechniques to generate the supplemental/pseudo training data set.

In exemplary embodiments, the generating of the supplemental trainingdata set includes generating one or more pseudo (synthetic) images fromthe original training data set, the one or more pseudo imagescorresponding or representing images of the target imaging modality. Thegenerated one or more pseudo images depict the same anatomical featuresas those present in the original training data set and the same groundtruths as those present in the original training data set as if theywere in images of the target imaging modality.

In exemplary embodiments, the original training data set includes imagedata of a first imaging modality, the first imaging modality beingdifferent from the target imaging modality.

In exemplary embodiments, the generating of the pseudo images includesat least a forward projection step or a forward and backward projectionstep to generate projection images from images of the original trainingdata set, and to reconstruct the projection images into a pseudo 3Dvolume or pseudo 2D scans, taking into consideration radiation doseattenuation as well as reconstruction artifacts specific to the targetimaging modality.

In exemplary embodiments, the generating of the pseudo images includesusing a neural network trained to predict a pseudo image of a targetimaging modality based on images of a first imaging modality.

The computer infrastructure 310 can be a treatment planning device, suchas the treatment planning device 300 shown in FIG. 6 , for example, thatis configured to perform any suitable number of treatment planning tasksor steps, such as segmentation, dose prediction, projection dataprediction, treatment plan 350 generation, etc. The treatment planningdevice 300 may include an image processing module 320 configured toperform segmentation to generate structure data 340A identifying variousanatomical structures, such as, but not limited to, the malignant tumor(i.e., the target), and any organs-at-risk (OAR), as shown in FIG. 7 ,for example. The structure data 340A may also identify other anatomicalstructures, such as other organs, tissues, bones, blood vessels, etc.The structure data 340A may also include any suitable data relating tothe contour, shape, size, and location of a patient's anatomy, themalignant tumor (i.e., the target), any organs-at-risk (OAR), and anyother anatomical structures. The treatment planning device 300 may alsoinclude a treatment planning module 330, by which dose data 340B, asshown in FIG. 7 , is determined specifying the radiation dose to bedelivered to the target and any other anatomical structure desired to beirradiated, and specifying the maximum allowable radiation dose that isallowed to be delivered to other anatomical structures, such as theOARs, for example. The treatment plan 350 may also contain any otheradditional data, such as prescription, disease staging, biologic orradiomic data, genetic data, assay data, past treatment or medicalhistory, or any combination thereof. The treatment plan 350 may alsotake into consideration constraints imposed on the treatment process bya radiation therapy system 100 show in FIG. 6 used for delivering theradiation to the patient 110.

When used as such, the computer infrastructure 310 is configured toautomatically generate contours using trained segmentation DNN models315 and to generate the treatment plan 350 for a patient to be executedusing a radiotherapy system, such as the radiotherapy system 100 shownin FIG. 6 , for example.

Alternatively, the computer infrastructure 310 can include an imageprocessing module, such as the image processing module 320 shown inFIGS. 5 and 6 , for example, to generate contours of anatomicalstructures of the patient on medical images using trained segmentationDNN models 315, and a treatment planning module 330 as shown in FIG. 6 ,for example, to generate the treatment plan 350 using the contouredmedical images.

Alternatively, the computer infrastructure 310 may provide the imageprocessing functions of the image processing module 320 to generatecontours of anatomical structures of a patient, with the treatmentplanning module 330 being separate from the computer infrastructure 310.

In an exemplary embodiment illustrated in FIGS. 5 and 6 , the computerinfrastructure 310 provides the image processing functions of the imageprocessing module 320 to generate contours of anatomical structures of apatient, and includes a computer system 312, a treatment planning module330, a network interface 311, and an input/output device 351 operativelycoupled to an input/output interface 210 of the radiotherapy system 100via network 600, for example.

The computer system 312 can implement one or more aspects of the processof FIGS. 8-22 . Although shown as a single module 312, the functionalityof module 312 can be implemented as a distributed system or otherwise.Moreover, although illustrated separately, the module 312 and thetreatment planning module 330 may be integrated together, for example,as a single module with both image processing and treatment planningfunctionality provided by memory 314, as separate parts of a commoncomputer system 312, or as separate parts of a common system (e.g., acentral or distributed processing system operating on a remote server).

The computer system 312 can include a bus 319 or other mechanism forcommunicating information between components. The computer system 312can also include a processor 313, for example, a general or specificpurpose processor (e.g., graphics processing unit (GPU)), coupled to bus319. The processor 313 can be a processor of a cloud-based system,and/or a processor of one or more network or Internet host servers. Thecomputer system 312 can include an input/output module 321, for example,a communication device such as network interface cards that provideaccess to network 600 to communicate with an image database 400 and/or401, and/or an image model database 500, and/or the radiation therapysystem 100, and/or an imaging device 700, and/or with input/output portsthat allow a user 351 to interact with the computer system 312, forexample via user input devices including a mouse, keyboard, display,etc., such as an interactive graphical user interface (GUI) 311.

The computer system 312 can also include a memory 314 that storesinformation and instructions to be executed by processor 313. The memory314 can be comprised of any combination of random access memory (RAM),read only memory (ROM), static storage such as a magnetic or opticaldisk, or any other type of computer readable media. For example,computer readable media may be any available media that can be accessedby processor 313 and can include both volatile and nonvolatile media,removable and non-removable media, and communication media.Communication media may include computer readable instructions, datastructures, program modules or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and includes anyinformation delivery media.

Memory 314 can store software modules that provide functionality whenexecuted by processor 313. The modules can include, for example, anoperating system 316 that can provide operating system functionality forthe computer system 312, one or more inference engines 318 configured toaccess and/or link to a plurality of machine learning models, such asthose saved in an external image model database 500 or the plurality ofmodels 315 saved in memory 314. The memory 314 also includes a dataaugmentation engine (DAE) 317, which can include a plurality of firstpseudo-image generations modules 317A and a plurality of secondpseudo-image generation modules 317B. The first pseudo-image generationmodules 317A can include hardware and/or software components that arecapable of executing algorithms to generate one or more pseudo imagesusing forward and backward projections steps, and hardware and/orsoftware components that are capable of executing algorithms to generateone or more pseudo images using only forward projection steps. Thesecond pseudo-image generation modules 317B can include hardware and/orsoftware components that are capable of executing algorithms accordingto one or more neural network models stored in the image model database500, and/or the memory 314, that are trained to generate pseudo imagesbased on input images. One or more of the neural network models aretrained to generate pseudo images of a second (target) imaging modalitybased on input images of a first imaging modality, the second imagingmodality being different from the first imaging modality.

The pseudo-image generation modules 317A, 317B can receive medical imagedata, including training data, from image database 400, or any otherinternal or external image database, or from the medical imaging device700 (i.e., planning medical images), or from the radiotherapy system 100via network 600, or via I/O 321, and generate one or more pseudo imagesbased on the received training data. The pseudo-image generation engines317A, 317B are further configured to receive an original set of trainingdata including one or more images of a first imaging modality andgenerate one or more pseudo images of a second, different imagingmodality based on the original set of training data. The pseudo-imagegeneration engines 317A, 317B are further configured to copy the anatomyand/or anatomical features related to an anomaly/disease and/or thecontours contained in the images of the original training data set tothe generated pseudo images, and therefore include the original data setanatomy and/or original features as ground truth features and/or theoriginal data set segmentation as ground truth contours in the generatedpseudo images. The generated pseudo image data set can be stored in astorage device of memory 314, or in the image database 400, or 401, orany other internal and/or external image database that are madeaccessible to the inference engines 318.

The inference engines 318 are modules that include hardware and/orsoftware components that are capable of executing algorithms accordingto the models stored in the image model database 500, and/or the models315 stored in the memory 314. The one or more inference engines 318 canreceive medical image data (training data including original and pseudoimages, projections data, or medical image(s) for inference) from imagedatabase 400, 401, or any other internal or external image database, orfrom the medical imaging device 700 (i.e., planning medical images), orfrom the radiotherapy system 100 via network 600, or via I/O 321, andgenerate contours for one or more anatomical structures in the receivedmedical images based on one or more image segmentation models 315 or anyother trained neural network stored in one of the model databases,and/or to detect and/or predict anatomical anomalies and/or diseases,such as but not limited to, a collapsed lung, for example, in thereceived medical images based on one or more diagnostic models or anyother trained neural network stored in one of the model databases.

The contours generated by the computer infrastructure 310 can beprocessed by the treatment planning module 330 and the generatedtreatment plan 350 can be executed using a radiotherapy system, such asthe radiotherapy system 100 shown in FIG. 6 , for example.

An exemplary radiation therapy system 100 can provide radiation to apatient 110 positioned on a treatment couch 112 and can allow for theimplementation of various radiation dose verification protocols. Theradiation therapy can include photon-based radiation therapy, particletherapy, electron beam therapy, or any other type of treatment therapy.

In an embodiment, the radiation therapy system 100 can include aradiation treatment device 101 such as, but not limited to, a LINACoperable to generate one or more beams of megavolt (MV) X-ray radiationfor treatment. The LINAC may also be operable to generate one or morebeams of kilovolt (kV) X-ray radiation, for example, for patientimaging. The system 100 has a gantry 102 supporting a radiationtreatment head 114 with one or more radiation sources 106 and variousbeam modulation elements, such as, but not limited to, flattening filter104 and collimating components 108. The collimating components 108 caninclude, for example, a multi-leaf collimator (MLC), upper and lowerjaws, and/or other collimating elements. The collimating components 108and/or the flattening filter 104 can be positioned within the radiationbeam path by respective actuators (not shown), which can be controlledby controller 200.

The gantry 102 can be a ring gantry (i.e., it extends through a full360° arc to create a complete ring or circle), but other types ofmounting arrangements may also be employed. For example, a static beam,or a C-type, partial ring gantry, or robotic arm can be used. Any otherframework capable of positioning the treatment head 114 at variousrotational and/or axial positions relative to the patient 110 may alsobe used.

In an embodiment, the radiation therapy device is a MV energy intensitymodulated radiation therapy (IMRT) device. The intensity profiles insuch a system are tailored to the treatment requirements of theindividual patient. The IMRT fields are delivered with MLC 108, whichcan be a computer-controlled mechanical beam shaping device attached tothe head 114 and includes an assembly of metal fingers or leaves. Foreach beam direction, the optimized intensity profile is realized bysequential delivery of various subfields with optimized shapes andweights. From one subfield to the next, the leaves may move with theradiation beam on (i.e., dynamic multi-leaf collimation (DMLC)) or withthe radiation beam off (i.e., segmented multi-leaf collimation (SMLC)).

Alternatively, or additionally, the radiation therapy device 101 can bea tomotherapy device, a helical tomotherapy device, or a simplifiedintensity modulated arc therapy (SIMAT) device, a volumetric modulatedarc therapy (VMAT) device, or a volumetric high-definition (or hyperarc)therapy (HDRT). In effect, any type of IMRT device can be employed asthe radiation therapy device 101 of system 100, and can also include anon-board volumetric imaging, which can be used to generate in-treatmentimage data generated during a treatment session.

Each type of radiation therapy device can be accompanied by acorresponding radiation plan and radiation delivery procedure.

The controller 200, which can be, but is not limited to, a graphicsprocessing unit (GPU), can include a computer with appropriate hardwaresuch as a processor, and an operating system for running varioussoftware programs and/or communication applications. The controller 200can include software programs that operate to communicate with theradiation therapy device 101, which software programs are operable toreceive data from external software programs and hardware. The computercan also include any suitable input/output (I/O) devices 210, which canbe adapted to allow communication between controller 200 and a user ofthe radiation therapy system 100, e.g., medical personnel. For example,the controller 200 can be provided with I/O interfaces, consoles,storage devices, memory, keyboard, mouse, monitor, printers, scanner, aswell as a departmental information system (DIS) such as a communicationand management interface (DICOM) for storing and transmitting medicalimaging information and related data and enabling the integration ofmedical imaging devices such as scanners, servers, workstations,printers, network hardware, etc.

Alternatively, or additionally, the I/O devices 210 can provide accessto one or more networks, such as network 600, for example, fortransmitting data between controller 200 and remote systems. Forexample, the controller 200 can be networked via I/O 210 with othercomputers and radiation therapy systems. The radiation therapy system100, the radiation treatment device 101, and the controller 200 cancommunicate with the network 600 as well as databases and servers, forexample, a dose calculation server (e.g., distributed dose calculationframework) and the treatment planning system 300. The controller 200 mayalso be configured to transfer medical image related data betweendifferent pieces of medical equipment.

The system 100 can also include a plurality of modules containingprogrammed instructions (e.g., as part of controller 200, or as separatemodules within system 100, or integrated into other components of system100), which instructions cause system 100 to perform different functionsrelated to adaptive radiation therapy or other radiation treatment. Forexample, the system 100 can include a treatment plan module operable togenerate the treatment plan for the patient 110 based on a plurality ofdata input to the system by the medical personnel, a patient positioningmodule operable to position and align the patient 110 with respect to adesired location, such as the isocenter of the gantry, for a particularradiation therapy treatment, an image acquiring module operable toinstruct the radiation therapy system and/or the imaging device toacquire images of the patient 110 prior to the radiation therapytreatment (i.e., pre-treatment/reference images used for treatmentplanning and patient positioning) and/or during the radiation therapytreatment (i.e., in-treatment session images), and to instruct theradiation therapy system 100 and/or the imaging device 101 or otherimaging devices or systems to acquire images of the patient 110.

The system 100 can further include a radiation dose prediction moduleoperable to predict a dose to be delivered to the patient 110 beforecommencement of the radiation treatment therapy, a dose calculationmodule operable to calculate the actual dose delivered to the patient110 during radiation therapy treatment, a treatment delivery moduleoperable to instruct the radiation therapy device 100 to deliver thetreatment plan to the patient 110, a correlation module operable tocorrelate the planning images with the in-treatment images obtainedduring radiation therapy, a computation module operable to reconstructthree-dimensional target volumes from in-treatment images, an analysismodule operable to compute displacement measurements, and a feedbackmodule operable to instruct the controller in real-time to stopradiation therapy based on a comparison of the calculated displacementwith a predetermined threshold value (range).

The system 100 can further include one or more contour generationmodules operable to generate contours of target volumes and otherstructures in pre-treatment (planning, reference) and in-treatment(treatment session) images, an image registration module operable toregister pre-treatment images with subsequent in-treatment images, adose calculation module operable to calculate accumulated dose, acontour propagation module operable to propagate a contour from oneimage to another, a contour verification module operable to verify agenerated contour, a registration deformation vector field generationmodule operable to determine deformation vector fields (DVFs) as aresult of an image deformation process. The system 100 can furtherinclude modules for electron density map generation, isodosedistribution generation, does volume histogram (DVH) generation, imagesynchronization, image display, treatment plan generation, treatmentplan optimization, automatic optimization parameter generation, updatingand selection, and adaptive directives and treatment informationtransfer. The modules can be written in the C or C++ programminglanguage, for example. Computer program code for carrying out operationsas described herein may be written in any programming language, forexample, C or C++ programming language.

Although the discussion of FIGS. 5 and 6 above has focused on the use ofthe computer infrastructure 310 with a radiotherapy system 100,embodiments of the disclosed subject matter are not limited thereto.Indeed, the computer infrastructure 310 may be provided as a separateindependent system for image analysis, may be integrated with an imagingmodality, may communicate with other medical treatment systems, or maybe integrated with other medical treatment systems. Accordingly,embodiments of the disclosed image processing module are not limited tothe specific configuration illustrated in FIG. 6 or limited to use withradiotherapy systems. Instead, in exemplary embodiments, the traineddiagnostic models 315 may be used to diagnose a patient.

Referring to FIGS. 8-20 , exemplary operations of the computerinfrastructure 310 to train and apply automatic segmentation DNN models315 will be described. FIGS. 10A-10D, and 11A-11B are schematicillustrations of the operation of a segmentation DNN models 315 during atraining phase 400 (400A-400D). FIGS. 12A-12C are schematicillustrations of the operation of a segmentation DNN model 315 during aninference phase 500. FIGS. 8 and 9 are process flow diagrams of varioussteps performed during training 400 and inference 500 phases, FIG. 13 isa generalized process flow diagram of various augmentation processesthat can be used to generate pseudo images to augment the originaltraining data set used in the training phase 500, and FIGS. 14-20 areschematic illustrations of pseudo training data set generation steps fortraining data augmentation underlying the training of the segmentationDNN models 315.

The generation of automatic segmentation in a patient image by thecomputer infrastructure 310 can begin with process S500 shown in FIG. 8. Process S500 can start with the training phase 400, which trains asegmentation DNN model 315 on an appropriate training data set. Thesegmentation DNN model 315 is trained so as to generate contours ofanatomical structures in patient images of a target imaging modality(i.e., CBCT images, for example).

In the training phase 400, the computer infrastructure 310 can have asetup 400A-400D as illustrated in FIGS. 10A-10D. The process S500 canbegin at S502, where an original training data set 402 of medical images404 is provided. The medical images of the original training data set,whether 2-D or 3-D images, can be of any suitable first imagingmodality, such as, but not limited to, X-ray, computer tomography (CT),cone beam computed tomography (CBCT), spiral CT, positron emissiontomography (PET), magnetic resonance imaging (MRI), functional MRI,single photon emission computed tomography (SPECT), optical tomography,ultrasound imaging, fluorescence imaging, radiotherapy portal imaging,or any combinations thereof. For example, image data may include aseries of 2-D images or slices, each representing a cross-sectional viewof the patient's anatomy. Alternatively or additionally, image data mayinclude volumetric or 3-D images of the patient, or a time series of 2-Dor 3-D images of the patient.

The original training data set 402 can also include desired output, inthe form of ground truth, such as contours 406 of one or more anatomicalstructures, that are verified by an expert or system user. For example,the original training data 402 can be user-generated throughobservations and experience to facilitate supervised learning and may beextracted from past images of previous patients. Preferences of the useror facility can thus be taken into account in the processing by virtueof the user-defined training data set 402.

In some embodiments, as shown in FIGS. 11A and 11B, the originaltraining data set 402 includes 3-D (CT) image(s) 404 and itscorresponding 3-D ground truth label map 406 that associates ananatomical structure to each of the voxels of the 3-D image(s). In someembodiments, the 3-D image(s) 404 may be divided into a sequential stackof adjacent 2-D images 404, and the 3-D ground truth label map caninclude sequential 2-D ground truth label maps, respectivelycorresponding to the sequential stack of adjacent 2-D images. Theoriginal training data set 402 can have images that have already beensegmented (i.e., contoured), where a ground truth label map provides aknown anatomical structure label for each pixel of a representativeimage slice of the original training image. In other words, pixels ofthe ground truth label map can be associated with known anatomicalstructures.

The original training data set 402 is next processed in S504 using apseudo-image generation approach 317, the specific steps thereofillustrated in detail in FIGS. 13-20 , to produce a supplementaltraining data set 410 (pseudo training data set) with properties thatprovide realistic image and segmentation data for the target imagingmodality. In embodiments, the target imaging modality can be differentfrom the imaging modality of the original training data set 402.

The pseudo training data set 410 can include pseudo (synthetic) images(scans) 412 that are generated from the original training data set 402.The generated pseudo images 412 provide a realistic model of theinteraction of image generating signals (X-rays, for example) with thepatient 110, while also providing a realistic patient model (i.e.,accurate contours of the anatomical structures, and/or accurate featuresrelating to a condition of an anatomical structure). The generatedpseudo images 412 also contain the same ground truth contours 406 as theoriginal training images 404, while depicted as if contained in imagesof the target imaging modality.

In exemplary embodiments, the generated pseudo images 412 include imagesof a second imaging modality, with the second imaging modality beingdifferent from the first imaging modality of the training data set 402.

In exemplary embodiments, the generated pseudo images 412 include imagesof a target imaging modality, with the target imaging modality beingdifferent from the first imaging modality of the training data set 402.

In the exemplary embodiment of FIGS. 11A and 11B, the pseudo trainingdata set 410 includes 3-D (CBCT) image(s) 412 and its corresponding 3-Dground truth label map 406 that associates an anatomical structure toeach of the voxels of the 3-D image(s). In some embodiments, the 3-Dimage(s) 412 may be divided into a sequential stack of adjacent 2-Dimages 412, and the 3-D ground truth label map 406 can includesequential 2-D ground truth label maps, respectively corresponding tothe sequential stack of adjacent 2-D images. The pseudo training dataset 410 can therefore have pseudo images 412 where a ground truth labelmap 406 provides a known anatomical structure label for each pixel of arepresentative image slice of the original training image 404. In otherwords, pixels of the ground truth label map can be associated with knownanatomical structures.

Optionally, the pseudo training data set 410 can be further processed asshown in FIGS. 10C and 10D to combine any two or more pseudo images 412from the pseudo training data set 410 using any suitableaugmentation/combination approach 360 to generate a second pseudotraining data set 410′ including the combined pseudo images 412′ withmodified properties that may improve the segmentation DNN model 315generalization.

Optionally, each of the original training data set 402, and the firstand second pseudo training data sets 410, 410′ can be furtherpre-processed using any suitable data augmentation approach (e.g.,rotation, flipping, translation, scaling, noise addition, cropping, orcombinations thereof) to produce additional training data sets withmodified properties that may further improve model generalization.

Optionally, the original training data set 402 and the first and secondpseudo training data sets 410, 410′ can include additional subsets. Forexample, data sets 402, 410 and 410′ can each include a validation setthat is used to track the quality of the segmentation DNN model 315during training thereof, but is not otherwise used as input to the model315 during training.

Alternatively or additionally, the original training data set 402, andthe first and second pseudo training data sets 410, 410′ can eachinclude a test subset that is only used after training to quantify thequality of the trained model 315 (e.g., accuracy, dice score) and/or toverify that the model 315 has not over-learned or under-learned thedata.

In training phase 400, the process S500 can proceed to S506, wheretraining of the segmentation DNN model 315 occurs. The segmentation DNNmodel 315 can be trained to output contours 414 (414A-414D) foranatomical structures in the medical images 404. The training step S506of the segmentation DNN model 315 is according to the process flow ofFIG. 9 , and can be represented by the layouts 400A-400D of FIGS.10A-10D, and 11A-11B.

In particular, at S516, the training data (S516A and/or S516B and/orS516C) is provided as input data to the input layer of the segmentationDNN model 315. In the exemplary embodiment of FIG. 10A, the providing ofthe training data includes the providing at S516A of the training datafrom the original training data set 402 and the providing at S516B ofthe pseudo data from the first pseudo training data set 410. In theexemplary embodiment of FIG. 10B, the providing of the training dataincludes the providing at S516B of only the pseudo training data fromthe first pseudo training data set 410. In the exemplary embodiment ofFIG. 10C, the providing of the training data includes the providing atS516A of the training data from the original training data set 402, theproviding at S516B of the pseudo training data from the first pseudotraining data set 410, and the providing at S516C of the pseudo trainingdata from the second pseudo training data set 410′. In the exemplaryembodiment of FIG. 10D, the providing of the training data includes theproviding at S516B of the pseudo training data from the first pseudotraining data set 410 and the providing at S516C of the pseudo data fromthe second pseudo training data set 410′.

At S526, the segmentation DNN model 315 processes the input data (i.e.,input images) by propagating through nodes of its hidden layers. AtS536, the segmentation DNN model 315 produces an output 420A-420D (e.g.,contours of anatomical structures 414A-414D) at its respective outputlayer, which output 420A-420D (414A-414D) can be compared to groundtruth contours 406 via a loss function 416 at S546. For example, lossfunction 416 can be mean-squared error, dice loss, cross entropy-basedlosses or any other loss function known in the art.

During the training S506, the segmentation DNN model 315 is givenfeedback 422A-422D (by loss function 416) on how well its output420A-420D matches the correct output 421A-421D. The aim of training S506is to train the segmentation DNN model 315 to perform automaticsegmentation of anatomical structures in the image(s) by mapping inputdata (i.e., contours in the original and pseudo images) to exampleoutput data (i.e., ground truth contours 406). In some embodiments,training S506 can involve finding weights that minimize the trainingerror (e.g., as determined by loss function 416) between ground truthcontours 406 and estimated contours 414A-414D generated by the deeplearning engine.

Once an iteration criteria is satisfied at S556 (e.g., loss function 416meets a predetermined threshold, a threshold number of iterations hasbeen reached, or no further improvement is seen between iterations), thesegmentation DNN model 315 is fixed at S576. Otherwise, the trainingS506 proceeds to S566, where the model 315 is modified, e.g., byadjusting parameters of the hidden layer nodes, in order to improve thematch between output 420A-420D and the desired output 421A-421D. Thetraining process S506 thus can iterate repeatedly until the desirediteration criteria is satisfied at S556.

Alternatively or additionally, some parameters can be defined andadjusted at S566 in order to only impact training S506 of segmentationDNN model 315 without otherwise affecting inference, such as, but notlimited to, loss function, hyper parameters (e.g., dropout,regularization), training data augmentation (e.g., to avoid overlearningand achieve better generalization), and preprocessing of input data(e.g., scaling, normalization).

In some embodiments, the providing of the training data set at S516 caninclude modifying the training data set to improve consistency thereof,for example, by processing the various medical images 404 of data set402 or the pseudo images 412 of data set 410 or the pseudo images 412′of data set 410′. However, such processing may be time-intensive as itwould require manual processing by the user, as well as re-training ofthe model 315 after each correction to see if the outputs 420A-420D areimproved.

Once the training phase 400 has completed, the process S500 can proceedto the inference phase 500, which uses the trained DNN model 315 toprocess medical image(s) of a patient to automatically segment (i.e.,contour) unknown anatomical structures shown therein. In inference phase500, the image processing module can have a setup 500A-500C asillustrated in FIGS. 12A-12C. The process 500 can begin at S508, where apatient data set 502 including one or more medical image(s) 504 areprovided to the trained segmentation DNN model 315. The image(s) of thepatient can be obtained using any target medical imaging modality, suchas, such as, but not limited to, X-ray, computer tomography (CT), conebeam computed tomography (CBCT), spiral CT, positron emission tomography(PET), magnetic resonance imaging (MRI), functional MRI, single photonemission computed tomography (SPECT), optical tomography, ultrasoundimaging, fluorescence imaging, radiotherapy portal imaging, or anycombinations thereof. For example, image data may include a series of2-D images or slices, each representing a cross-sectional view of thepatient's anatomy. Alternatively or additionally, image data may includevolumetric or 3-D images of the patient, or a time series of 2-D or 3-Dimages of the patient. The target imaging modality can be different fromthe first imaging modality of the original training data set 402. Thetarget imaging modality can be the same as the imaging modality of thegenerated pseudo images 412 of the pseudo training data set 410 and/orthe generated pseudo images 412′ of the pseudo training data set 410′.

In an exemplary embodiment, as shown in FIG. 12C, the patient image dataset 502 includes 3-D (CBCT) image(s) 504 including anatomical structuresto be delineated. In some embodiments, the 3-D CBCT image(s) 504 may bedivided into a sequential stack of adjacent 2-D images 502.

Process 500 can then proceed to S510, where the medical image(s) 504 areprocessed by the trained segmentation DNN model 315. The segmentationDNN model 315 thus outputs contour data 506 based on its training. Insome embodiments, the contour data 506 may be combined with the originalmedical image(s) at S512, such that the contours 506 are overlaid on thecorresponding anatomical structures in the image 504, for example, forvisualization by a user or for use in radiation treatment planning.Alternatively, the segmentation DNN model 315 may directly produce thecontours 506 on the medical images 504 as an output data set 508 withoutseparate combination step S512.

Various modifications of the layouts and processes illustrated in FIGS.10A-10D, 11A-11B, and 12A-12C are also possible according to one or morecontemplated embodiments. For example, in some embodiments, thesegmentation DNN model 315 may directly produce the contours on themedical images as an output, and the algorithm can directly modify thecontours on the medical images, without necessarily requiring a separatecombination step S512.

Alternatively or additionally, in some embodiments, non-imaginginformation can be used along with the medical image(s) of the patientin the inference phase. The non-imaging information can be provided tothe trained segmentation DNN model 315 for use in generating the contourdata 506.

Although a single segmentation DNN model 315 is illustrated in FIGS.10A-10D, 11A-11B, and 12A-12C, and discussed with respect to FIGS. 8 and9 , embodiments of the disclosed subject matter are not limited thereto.For example, contours can be generated by employing one or moresegmentation DNN models, working separately or in unison. For example,outputs from separate segmentation DNN models can be combined togetherto form a singular output of contour data. In such a configuration, eachsegmentation DNN model may have different operating characteristics,such as a different input image resolution, different hidden layerconfiguration, or different underlying neural network models (e.g., CNNversus RNN, etc.). Other variations and modifications will be readilyapparent to one of ordinary skill in the art.

Turning to the process of augmenting the original training data set 402to obtain the supplemental training data set 410, FIG. 13 illustratesexemplary processes that can be applied in S504 for generating thepseudo image(s) 412 of the pseudo training data set 410. As shown inFIG. 13 , in S1000, the data augmentation engine 317receives/obtains/accesses the original training data set 402 of medicalimages 404 including the contours 406 or anatomical features related toan anomaly/disease, etc. as ground truths. The data augmentation engine317 is configured to apply a plurality of data augmentation processes,including applying different pseudo-image generating algorithms andprocess steps to generate pseudo images 412 of second imaging modalityusing the images 404 of the training data set 402 and the segmentationdata (i.e., the contours) and/or anatomical features 406 containedtherein. In exemplary embodiments, the user 351 can choose whichpseudo-image generating process to apply. Alternatively, the dataaugmentation engine 317 may be configured to automatically determine andapply a suitable data augmentation process.

One of the options to generate the pseudo image(s) 412 of the pseudotraining data set 410 is to apply a forward projection step coupled witha backward projection step as shown in steps S1001-S1004. Another optionis to apply a forward projection step coupled with a neural network asshown in steps S1101-S1105. Another option is to apply a neural networkas shown in steps S1111-S1114 to generate the pseudo images 412.

For the first option, at S1001, a forward projection algorithm isapplied on the original training data set 402 to generate projectionimages (i.e., digitally reconstructed radiographs DRRs) 608 (608A-608N)from the original training images 404, as shown in FIGS. 14-16 , forexample.

To generate the projection images 608, a 3D array of voxels 405 (i.e.,volume image data) from the set of training images 404 of the originaltraining data set 402 is obtained. The 3D array of voxels 405 representsthe original 3D images or the 2D image slices 404 viewed as athree-dimensional array. Then an algorithm can be applied to simulate ahypothetical radiation beam 604, such as, but not limited to X-rays,from a virtual radiation source 602 centered on a point a distance awayfrom the volume 405 centroid, passing through the volume 405 atdifferent source positions (A-N) and corresponding projection angles(α1-αn), and captured on a plane of a virtual detector 607, as shown inFIGS. 14-15 , also positioned at a distance away from the volume 405centroid opposite the source 602. The line integrals over the Hounsfeldvalues across the rays through the voxel volume 405 construct each ofthe projection images 608A-608N in S1002, as shown in FIG. 16 . Thegenerated 2D projection images 608 (i.e., DRRs) represent projections ofthe anatomical structures and ground truths (contours and/or anatomicalfeatures) 406 of the 3D volume image 405.

A back-projection algorithm is next applied in S1003 to accumulate(i.e., reconstruct) the generated projection images 608 onto a 3D volume412A, taking into consideration the angles α from which the projectionimages 608 were acquired, as shown in FIG. 17 . The reconstructed 3Dvolume 412A is a computer-generated image (i.e., a pseudo 3D volumetricimage) representing a 3D image acquired in a target imaging modality,while based on actual image data acquired from images of the firstimaging modality (i.e., original training images 404). The generatedpseudo 3D volumetric image 412A contains the anatomical structures andground truth contours and/or features 406 of the corresponding originaltraining images 404.

The pseudo 3D volumetric image 412A obtained in S1003 can be furtherrepresented as a series of 2-D images or slices 412, each representing across-sectional view of the patient's anatomy, as shown in FIG. 17 .

In order to generate a pseudo 3D volumetric image 412A (and/or 2D imagescans of pseudo images 412) that provides a realistic model of theinteraction of image generating signals (X-rays, for example) with thepatient in the target imaging modality, while also providing a realisticpatient model, the forward and backward projection algorithms applied togenerate and reconstruct the projection images 608, accurately model thebehavior of photons as they pass and scatter 606 through the volumetricimages 405, accurately calculate the radiation dose on the virtualdetector 607 pixels, and establish an accurate estimate of the actualimaging geometry corresponding to the generated projection images 608.In other words, the forward and backward projection algorithmsaccurately simulate the conditions present in the target imagingmodality, and account for artifacts that are present in that imagingmodality.

The algorithm applied in S1001 is a forward projection algorithm thattakes into account the specific source and detector geometry used in thetarget imaging modality, the geometry of the radiation beam in thetarget imaging modality, as well as a plurality of particle-matterinteractions, such as but not limited to scatter and penumbra, presentin the target imaging modality, so as to accurately model the behaviorof photons as they pass and scatter 606 through the volumetric images405, accurately calculate the radiation dose, and establish an accurateestimate of the actual imaging geometry corresponding to the generatedprojection images 608.

In an exemplary embodiment, the forward projection algorithm can includeone or more parameters for scatter correction, such as but not limitedto, parameters to account for scatter through the patient holding device(i.e., patient couch), parameters to account for scatter throughdifferent material densities, parameters to account for scatter fromelements of an imaging device used in generating images of the targetimaging modality, parameters to account for artifacts introduced bymovement of the source and/or the detector, and parameters to accountfor movement of the patient during imaging using the target imagingmodality.

In exemplary embodiments, the forward projection algorithm furthersimulates artifacts that occur due to motion during the imageacquisition. Motion may be simulated by deforming the 3D volume image405 for each projection angle α before forward projection.

Alternatively, the forward projection algorithm may simulate artifactsdue to motion during the image acquisition by overlaying a moving objecton the 3D volume image 405 for each projection angle α before forwardprojection.

Additionally, the forward projection algorithm may further includeadditional parameters to correct for artifacts introduced by image lag,detector scatter, body scatter and beam hardening, for example.

In exemplary embodiments, the forward projection algorithm is furtherconfigured to generate primary projection images and scatter images, thescatter images representing the portion of the projection image 608generated by the scattered rays 606.

In exemplary embodiments, the forward projection algorithm is furtherconfigured to remove the scatter images from the primary projectionimages.

In exemplary embodiments, the projection images obtained in S1002 can bethe projection images 608 from which scatter images have been removed.

The back-projection algorithm also takes into account one or more of thesource and detector geometry and the geometry of the radiation beam inthe target imaging modality. In exemplary embodiments, theback-projection algorithm can be an iterative reconstruction algorithmthat includes parameters to correct for artifacts introduced by imagelag, detector scatter, body scatter and beam hardening, for example.

In exemplary embodiments, the back-projection algorithm can furtherinclude parameters that correct for noise and artifacts introduced bypatient motion, for example.

As an alternative option, pseudo images 412 may be generated by applyinga forward projection algorithm at S1101 on the images 404 of theoriginal training data set 402 to obtain projection image data at S1102,and instead of using a back-projection step to reconstruct the pseudo 3Dimage, the projection image data obtained in S1102 can be used as inputfor training a segmentation DNN model 315A to generate an output, suchas contours 414E for the anatomical structures in the projection images608, as shown in FIG. 18 , for example. The forward projection algorithmcan be the same as the forward projection algorithm used in the firstoption. Namely, the forward projection algorithm is such that itaccounts for the artifacts of the target imaging modality.

The steps taken in the training of the segmentation DNN model 315A couldalso follow the general model training steps as shown in FIGS. 2D, 3 and4 . For example, for the training of the segmentation DNN model 315A,the projection data, including the projection images 608 obtained usinga forward projection algorithm, and the ground truth contours 406contained therein, is provided as input data to the input layer of thesegmentation DNN model 315A. The segmentation DNN model 315A processesthe projection images 608 by propagating through nodes of its hiddenlayers. The segmentation DNN model 315A produces an output 414E (e.g.,contours of anatomical structures) at its respective output layer, whichoutput 414E can be compared to ground truth contours 406 via a lossfunction 416. For example, loss function 416 can be mean-squared error,dice loss, cross entropy-based losses or any other loss function knownin the art.

During the training, the segmentation DNN model 315A is given feedbackon how well its output 414E matches the correct output 406. The aim oftraining is to train the segmentation DNN model 315A to performautomatic segmentation of anatomical structures in the projectionimage(s) 608 by mapping input data to example output data (i.e., groundtruth contours 406). In some embodiments, training can involve findingweights that minimize the training error (e.g., as determined by lossfunction 416) between ground truth contours 406 and estimated contours414E generated by deep learning engine.

Once an iteration criteria is satisfied (e.g., loss function 416 meets apredetermined threshold, a threshold number of iterations has beenreached, or no further improvement is seen between iterations), thesegmentation DNN model 315A is fixed. Otherwise, the training continuesby modifying the model 315A, e.g., by adjusting parameters of the hiddenlayer nodes, in order to improve the match between output 414E and thedesired output 406. The training process thus can iterate repeatedlyuntil the desired iteration criteria is satisfied.

The so trained segmentation DNN model 315A can then be applied in S1104on the images 404 of the original training data set 402 to obtaincontour data, which, when combined with the images 404 of the originaltraining data set 402 in S1105, gives as output, pseudo 2D images oftarget imaging modality.

As another alternative option, the pseudo images 412 can be generated asshown in process steps S1111-S1114. In S1111, a trained neural networkmodel 315B is applied on the images 404 of the original training dataset 402. The trained network model 315B can be a convolutional neuralnetwork (CNN) model stored in the storage database of memory 312 or theexternal model storage database 500, for example, that is trained toautomatically generate a pseudo image(s) of a second imaging modalityfrom images of a first imaging modality. When applied on the images 404of the original training data set 402 of the first imaging modality, thetrained CNN model 315B generates, as output, pseudo image data of thesecond imaging modality in S1112, as shown in FIGS. 19 and 20 . Theobtained pseudo image data of the second imaging modality combined withmedical images of the first imaging modality 404 in S1113, outputs a setof pseudo images 412 of second imaging modality that include the groundtruth contours 406 of the original training data set 402.

Although the processes illustrated in FIGS. 8-20 are specific to thetraining and inference phases of a segmentation DNN model, embodimentsof the disclosed subject matter are not limited thereto. For example,instead of a segmentation DNN model 315, a diagnostic model 315C can betrained (400E, 400F) using an augmented data set to recognize ananatomic anomaly and/or condition in the training data set 402, andassociate the anomaly/condition with a particular disease, such as, butnot limited to a collapsed lung, for example, as shown in FIGS. 21A-21Band 22 . In such a scenario, the original training data set 402 does notinclude contours as ground truths, and the pseudo images 412 may notinclude the contours from the original training images. Instead, theoriginal images 404 of the training data set 402 may include one or moreanatomical features 407 associated with a particular disease as groundtruth, and the generated pseudo images 412 include these features. Thediagnostic model is trained to detect (414E, 414F) these features inmedical images (FIGS. 21A-21B), so as to recognize the presence of aparticular condition 507 related to disease, for example, in medicalimages 504 of a patient during inference 500D (FIG. 22 ).

It will therefore be appreciated that methods are described wherein atraining data set is augmented with pseudo images that accuratelyreflect the ground truths in imaging modalities of the patient images inthe inference phase.

It will also be appreciated that methods are described to train asegmentation model to approximate contours in patient images that aredifficult to otherwise segment.

It will also be appreciated that methods are described to train adiagnostic model to detect features in medical images that areassociated with a particular disease and/or anatomicalanomaly/abnormality.

It will also be appreciated that methods are described comprising:training a neural network model using a first data set of medical imagesand a second data set of medical images, the medical images of the firstdata set including medical images of a first imaging modality, each ofthe medical images of the first data set including a contour of ananatomical structure therein, the medical images of the second data setincluding pseudo medical images of a second imaging modality, the pseudomedical images of the second data set being generated from correspondingmedical images of the first data set, each of the pseudo medical imagesof the second data set including therein the contour of the anatomicalstructure contained in a corresponding medical image of the first dataset, and training the neural network model using at least the medicalimages of the second data set and the copied contours therein toapproximate the contour of the anatomical structure.

It will also be appreciated that methods are described comprising:augmenting a first data set of medical images with a second data set ofmedical images, the medical images of the first data set being of afirst imaging modality and the medical images of the second data setbeing pseudo medical images of a second imaging modality, and training amachine learning model using at least the pseudo medical images of thesecond data set and features included therein, wherein the pseudo imagesof the second data set are generated from corresponding medical imagesof the first data set, and wherein the features in the pseudo images arefeatures that are included in corresponding medical images of the firstdata set.

The machine learning model can be a segmentation model, and each of themedical images of the first data set can include a contour of theanatomical structure therein, each of the pseudo medical images of thesecond data set can include the contour of the anatomical structurecontained in corresponding medical images of the first data set, and thetraining of the segmentation neural network model includes using atleast the pseudo medical images of the second data set and the contourscontained therein.

Alternatively, or additionally, the machine learning model can be adiagnostic model, each of the medical images of the first data set caninclude a feature of an anatomical condition therein, each of the pseudomedical images of the second data set can include the feature of theanatomical condition contained in corresponding medical images of thefirst data set, and the training of the diagnostic model includes usingat least the pseudo medical images of the second data set and thefeatures of the anatomical condition contained therein.

It will also be appreciated that systems are described comprising: oneor more data storage devices storing at least one machine learningmodel, the machine learning model having been trained to approximate acontour of an anatomical structure or to predict an anatomicalcondition, and one or more processors operatively coupled to the one ormore data storage devices and configured to employ the at least onemachine learning model to process one or more medical images of a targetimaging modality of a patient to generate one or more contours ofanatomical structures in the medical images of the patient or to predictan anatomical condition from the medical images of the patient, whereinthe one or more processors are further configured to train the machinelearning model using a first data set of medical images and a seconddata set of medical images, the medical images of the first data setincluding medical images of a first imaging modality, and the medicalimages of the second data set including pseudo medical images of thetarget imaging modality, the one or more processors being furtherconfigured to generate the pseudo medical images of the second data setfrom corresponding medical images of the first data set.

It will also be appreciated that systems are described, comprising: oneor more data storage devices storing at least one neural network model,the neural network model having been trained to approximate a contour ofan anatomical structure, and one or more processors operatively coupledto the one or more data storage devices and configured to employ the atleast one neural network model to process one or more medical images ofa target imaging modality of a patient to generate one or more contoursof anatomical structures in the medical images of the patient, whereinthe one or more processors are further configured to train the neuralnetwork model to approximate contours of anatomical structures using afirst data set of medical images and a second data set of medicalimages, the medical images of the first data set including medicalimages of a first imaging modality, and the medical images of the seconddata set including pseudo medical images of the target imaging modality,the one or more processors being further configured to generate thepseudo medical images of the second data set from corresponding medicalimages of the first data set.

Each of the medical images of the first data set can include a contourof an anatomical structure therein, and the one or more processors areconfigured to replicate the contours of the medical images of the firstdata set in corresponding pseudo images of the second data set.

The one or more processors can be further configured to generate thepseudo images of the second imaging modality using one of a forwardprojection technique, a forward projection coupled with a backwardprojection technique, and a trained neural network technique.

The forward projection technique can include generating projectionimages from the medical images of the first data set.

The generating of the projection images can include simulating radiationpassing through the volumetric image, wherein the simulating can includedetermining radiation attenuation based on a plurality of firstparameters including radiation beam geometry, radiation source geometry,radiation detector geometry, and radiation interaction with matter.

The generating of the projection images can include generating avolumetric image from 2D training image slices and simulating radiationpassing through the volumetric image, wherein the simulating can includedetermining radiation attenuation based on a plurality of firstparameters including radiation beam geometry, radiation source geometry,radiation detector geometry, and radiation interaction with matter.

The generating of the projection images can further include determiningradiation dose distributed through the volumetric image data andgenerating pseudo-dose distribution matrices based on the calculatedradiation dose matrices. The generated pseudo-dose distribution matricescan also be used as input data to train a deep neural network.

The forward projecting step can further include simulatingreconstruction artifacts related to the second imaging modality.

The reconstruction artifacts can include artifacts related to motionoccurring during imaging using the second imaging modality.

The simulating can include deforming the volumetric image before thegenerating of projection images and/or overlaying a moving object on thevolumetric image.

The forward and backward projection technique can include reconstructingthe projection images into a pseudo volumetric image of the medicalimages of the second data set, wherein the reconstructing of theprojection images can include accumulating the projection images onto apseudo 3D volume, and generating 2D pseudo medical images of the seconddata set from the pseudo volumetric image.

The accumulating may be based on a plurality of second parametersincluding radiation source geometry and radiation detector geometry.

The generating of the pseudo images of the second data set can be doneusing a neural network trained to predict a pseudo image of a secondimaging modality based on an image of a first imaging modality.

In embodiments, the system can further comprise a radiotherapy deviceconfigured to deliver radiation treatment to a patient, wherein the oneor more processors are further configured to control the radiotherapydevice to irradiate the patient according to a treatment plan based atleast on the one or more medical images of the patient and the generatedcontours.

The one or more processors are configured to train the neural networkmodel by an iterative process and calculate a loss function after eachiteration; and receive input to modify the loss function prior to orduring the iterative process to change the contour of the anatomicalstructure.

It will also be appreciated that a non-transitory computer-readablestorage medium upon which is embodied a sequence of programmedinstructions is also described, and a computer processing system thatexecutes the sequence of programmed instructions embodied on thecomputer-readable storage medium to cause the computer processing systemto train a neural network model to approximate contours of anatomicalstructures using a first data set of medical images and a second dataset of medical images, the medical images of the first data setincluding medical images of a first imaging modality, and the medicalimages of the second data set including pseudo medical images of asecond imaging modality, generate the pseudo medical images of thesecond data set from corresponding medical images of the first data set,and process one or more medical images of a patient using the trainedneural network model to generate one or more contours of anatomicalstructures in the medical images of the patient.

It will be further appreciated that a non-transitory computer-readablestorage medium upon which is embodied a sequence of programmedinstructions is also described, and a computer processing system thatexecutes the sequence of programmed instructions embodied on thecomputer-readable storage medium to cause the computer processing systemto train a machine learning model using a first data set of medicalimages and a second data set of medical images, the medical images ofthe first data set including medical images of a first imaging modality,and the medical images of the second data set including pseudo medicalimages of a second imaging modality, generate the pseudo medical imagesof the second data set from corresponding medical images of the firstdata set, and process one or more medical images of a patient using thetrained machine learning model, wherein the execution of the sequence ofprogrammed instructions further causes the computer processing system togenerate the pseudo images of the second imaging modality using one of aforward projection technique, a forward projection coupled with abackward projection technique, wherein the forward projection techniqueincludes generating projection images from the medical images of thefirst data set by simulating radiation passing through volumetric imagesof the first data set, and wherein the backward projection techniqueincludes reconstructing the projection images into a pseudo volumetricimage of the medical images of the second data set, wherein thereconstructing of the projection images includes accumulating theprojection images onto a pseudo 3D volume, the forward projecting stepfurther including simulating acquisition inaccuracies related to thesecond imaging modality, the acquisition inaccuracies includinginaccuracies related to motion occurring during imaging using the secondimaging modality, wherein the accumulating is based on a plurality ofsecond parameters including radiation source geometry and radiationdetector geometry, and wherein the simulating includes determiningradiation attenuation based on a plurality of first parameters includingradiation beam geometry, radiation source geometry, radiation detectorgeometry, and radiation interaction with matter.

It will be appreciated that the aspects of the disclosed subject mattercan be implemented, fully or partially, in hardware, hardware programmedby software, software instruction stored on a computer readable medium(e.g., a non-transitory computer readable medium), or any combination ofthe above. For example, components of the disclosed subject matter,including components such as a controller, module, model, neuralnetwork, or any other feature, can include, but are not limited to, apersonal computer or workstation or other such computing system thatincludes a processor (e.g., graphics processing unit), microprocessor,microcontroller device, or is comprised of control logic includingintegrated circuits such as, for example, an application specificintegrated circuit (ASIC). Features discussed herein can be performed ona single or distributed processor (single and/or multi-core), bycomponents distributed across multiple computers or systems, or bycomponents co-located in a single processor or system. For example,aspects of the disclosed subject matter can be implemented via aprogrammed general purpose computer, an integrated circuit device (e.g.,ASIC), a digital signal processor (DSP), an electronic device programmedwith microcode (e.g., a microprocessor or microcontroller), a hard-wiredelectronic or logic circuit, a programmable logic circuit (e.g.,programmable logic device (PLD), programmable logic array (PLA),field-programmable gate array (FPGA), programmable array logic (PAL)),software stored on a computer-readable medium or signal, an opticalcomputing device, a networked system of electronic and/or opticaldevices, a special purpose computing device, a semiconductor chip, asoftware module or object stored on a computer-readable medium orsignal.

When implemented in software, functions may be stored on or transmittedover as one or more instructions or code on a computer-readable medium.The steps of any process, method, or algorithm disclosed herein may beembodied in a processor-executable software module, which may reside ona computer-readable medium. Instructions can be compiled from sourcecode instructions provided in accordance with a programming language.The sequence of programmed instructions and data associated therewithcan be stored in a computer-readable medium (e.g., a non-transitorycomputer readable medium), such as a computer memory or storage device,which can be any suitable memory apparatus, such as, but not limited toread-only memory (ROM), programmable read-only memory (PROM),electrically erasable programmable read-only memory (EEPROM),random-access memory (RAM), flash memory, disk drive, etc.

As used herein, computer-readable media includes both computer storagemedia and communication media, including any medium that facilitatestransfer of a computer program from one place to another. Thus, astorage media may be any available media that may be accessed by acomputer. By way of example, and not limitation, such computer-readablemedia may comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage or other magnetic storage devices, or anyother medium that may be used to carry or store desired program code inthe form of instructions or data structures and that may be accessed bya computer.

Also, any connection is properly termed a computer-readable medium. Forexample, if the software is transmitted from a website, server, or otherremote source using a transmission medium (e.g., coaxial cable, fiberoptic cable, twisted pair, digital subscriber line (DSL), or wirelesstechnologies such as infrared, radio, and microwave), then thetransmission medium is included in the definition of computer-readablemedium. Moreover, the operations of any process, method, or algorithmdisclosed herein may reside as one of (or any combination of) or a setof codes and/or instructions on a machine readable medium and/orcomputer-readable medium, which may be incorporated into a computerprogram product.

One of ordinary skill in the art will readily appreciate that the abovedescription is not exhaustive, and that aspects of the disclosed subjectmatter may be implemented other than as specifically disclosed above.Indeed, embodiments of the disclosed subject matter can be implementedin hardware and/or software using any known or later developed systems,structures, devices, and/or software by those of ordinary skill in theapplicable art from the functional description provided herein.

In this application, unless specifically stated otherwise, the use ofthe singular includes the plural, and the separate use of “or” and “and”includes the other, i.e., “and/or.” Furthermore, use of the terms“including” or “having,” as well as other forms such as “includes,”“included,” “has,” or “had,” are intended to have the same effect as“comprising” and thus should not be understood as limiting.

Any range described herein will be understood to include the endpointsand all values between the endpoints. Whenever “substantially,”“approximately,” “essentially,” “near,” or similar language is used incombination with a specific value, variations up to and including 10% ofthat value are intended, unless explicitly stated otherwise.

Many alternatives, modifications, and variations are enabled by thepresent disclosure. While specific examples have been shown anddescribed in detail to illustrate the application of the principles ofthe present invention, it will be understood that the invention may beembodied otherwise without departing from such principles. For example,disclosed features may be combined, rearranged, omitted, etc. to produceadditional embodiments, while certain disclosed features may sometimesbe used to advantage without a corresponding use of other features.Accordingly, Applicant intends to embrace all such alternative,modifications, equivalents, and variations that are within the spiritand scope of the present invention.

1. A method for generating pseudo images of a target imaging modality,comprising: selecting a pseudo-image generating process from a set ofpseudo-image generating processes; and applying the selectedpseudo-image generating process on a first data set of medical images,the medical images of the first data set including segmentation dataand/or anatomical features related to an anomaly or disease, the medicalimages of the first data set being of a first imaging modality, theselecting of the pseudo-image generating process being by a user or anaugmentation engine, and the applying of the selected pseudo-imagegenerating process includes applying one or more algorithms and processsteps configured to generate a second data set of medical images usingthe medical images of the first data set and the segmentation dataand/or anatomical features contained therein, the medical images of thegenerated second data set including pseudo medical images of the targetimaging modality that is different from the first imaging modality ofthe medical images of the first data set.
 2. The method of claim 1,wherein the set of pseudo-image generating processes includes at least aforward projection process, a backward projection process, and a neuralnetwork process.
 3. The method of claim 2, wherein applying the forwardprojection process includes generating projection images from themedical images of the first data set, wherein applying a backwardprojection process includes reconstructing the generated projectionimages into a pseudo volumetric image of the medical images, and whereinapplying a neural network process includes using a neural networktrained to predict pseudo images of the target imaging modality based onimages of the first imaging modality.
 4. The method of claim 3, whereinthe generating of the pseudo images of the second data set includesapplying a forward projection process combined with a backwardprojection process.
 5. The method of claim 4, comprising: applying aforward projection algorithm on the medical images of the first data setto obtain projection images; applying a back-projection algorithm on thegenerated projection images to accumulate the generated projectionimages onto a three-dimensional volume, the three-dimensional volumebeing a pseudo three-dimensional volumetric image representing athree-dimensional image of the target imaging modality; and representingthe pseudo three-dimensional volumetric image as a series of pseudotwo-dimensional images or image slices.
 6. The method of claim 5,wherein the obtaining of the projection images includes: obtaining athree-dimensional array of voxels from the medical images of the firstdata set, the array of voxels representing the medical images of thefirst data set viewed as a three-dimensional voxel volume; applying analgorithm to simulate a hypothetical radiation beam from a virtualradiation source passing through the three-dimensional voxel volume atdifferent radiation source positions and corresponding projection anglesand captured on a plane of a virtual detector; and generating lineintegrals across the radiation beams traversing the three-dimensionalvoxel volume to construct each of the projection images, wherein theprojection images represent projections of segmentation data and/oranatomical features of the three-dimensional voxel volume.
 7. The methodof claim 6, wherein the forward projection algorithm takes into accountradiation source and detector geometry used in the target imagingmodality, geometry of the radiation beam in the target imaging modality,and one or more particle-matter interactions in the target imagingmodality, in order to accurately simulate conditions present in thetarget imaging modality.
 8. The method of claim 7, wherein the one ormore particle interactions include scatter and penumbra present in thetarget imaging modality, and wherein the forward projection algorithmincludes parameters for scatter correction.
 9. The method of claim 5,wherein the accumulating of the generated projection images takes intoconsideration angles from which the projection images were acquired, andthe generated three-dimensional pseudo volumetric image contains thesegmentation data and/or anatomical features of the medical images ofthe first data set.
 10. The method of claim 9, wherein theback-projection algorithm accurately models behavior of radiation beamspassing and scattering through the three-dimensional pseudo volumetricimage to accurately simulate the conditions present in the targetimaging modality.
 11. The method of claim 10, wherein theback-projection algorithm includes an iterative reconstruction algorithmincluding parameters to correct artifacts introduced by one or more ofimage lag, detector scatter, radiation beam hardening, motion.
 12. Themethod of claim 3, wherein the generating of the pseudo images of thesecond data set includes applying a forward projection process combinedwith a neural network process.
 13. The method of claim 12, comprising:applying a forward projection algorithm on the medical images of thefirst data set to obtain projection images; inputting the projectionimages into a neural network model to train the neural network model togenerate an output, the output including segmentation data and/oranatomical features of the projection images; and applying the trainedneural network model on the medical images of the first data set toobtain segmentation and/or anatomical features data of the targetimaging modality; and combining the medical images of the first data setwith the obtained segmentation and/or anatomical features data to obtainpseudo images of the target imaging modality.
 14. The method of claim 3,wherein the generating of the pseudo images of the second data setincludes applying a neural network process.
 15. The method of claim 14,comprising: selecting a neural network process trained to automaticallygenerate a pseudo image of a target imaging modality from an image of afirst imaging modality; applying the trained neural network process onthe medical images of the first data set to generate pseudo image dataof the target imaging modality; combining the medical images of thefirst data set with the generated pseudo image data of the targetimaging modality to obtain pseudo images of the target imaging modality.16. A system comprising: one or more data storage devices storing a setof pseudo-image generating processes; and one or more processorsconfigured to: receive a first data set of medical images includingsegmentation data and/or anatomical features related to an anomaly ordisease, the medical images of the first data set being of a firstimaging modality, select a pseudo-image generating process to be appliedon the medical images of the first data set from the set, and apply theselected pseudo-image generating process on the medical images of thefirst data set to generate a second data set including pseudo-images ofa target imaging modality, wherein the applying the selectedpseudo-image generating process includes applying one or more algorithmsand process steps configured to generate the second data set of medicalimages using the medical images of the first data set and thesegmentation data and/or anatomical features contained therein, whereinthe target imaging modality of the pseudo-images of the second data setis different from the first imaging modality of the medical images ofthe first data set.
 17. The system of claim 16, wherein the set ofpseudo-image generating processes includes at least a forward projectionprocess, a backward projection process, and a neural network process,wherein applying the forward projection process includes generatingprojection images from the medical images of the first data set,applying a backward projection process includes reconstructing thegenerated projection images into a pseudo volumetric image of themedical images, applying a neural network process includes using aneural network trained to predict pseudo images of the target imagingmodality based on images of the first imaging modality, wherein thegenerating of the pseudo images of the second data set includes one ofapplying a forward projection process combined with a backwardprojection process, applying a forward projection process combined witha neural network process; and applying a trained neural network process.18. The system of claim 17, wherein the applying a forward projectionprocess combined with a backward projection process includes: applying aforward projection algorithm on the medical images of the first data setto obtain projection images; applying a back-projection algorithm on thegenerated projection images to accumulate the generated projectionimages onto a three-dimensional volume, the three-dimensional volumebeing a pseudo three-dimensional volumetric image representing athree-dimensional image of the target imaging modality; and representingthe pseudo three-dimensional volumetric image as a series of pseudotwo-dimensional images or image slices.
 19. The system of claim 17,wherein applying a forward projection process combined with a neuralnetwork process includes: applying a forward projection algorithm on themedical images of the first data set to obtain projection images;inputting the projection images into a neural network model to train theneural network model to generate an output, the output includingsegmentation data and/or anatomical features of the projection images;and applying the trained neural network model on the medical images ofthe first data set to obtain segmentation and/or anatomical featuresdata of the target imaging modality; and combining the medical images ofthe first data set with the obtained segmentation and/or anatomicalfeatures data to obtain pseudo images of the target imaging modality.20. The system of claim 17, wherein applying a trained neural networkprocess includes: selecting a neural network process trained toautomatically generate a pseudo image of a target imaging modality froman image of a first imaging modality; applying the trained neuralnetwork process on the medical images of the first data set to generatepseudo image data of the target imaging modality; and combining themedical images of the first data set with the generated pseudo imagedata of the target imaging modality to obtain pseudo images of thetarget imaging modality.